Context:¶
Typically e-commerce datasets are proprietary and consequently hard to find among publicly available data. This is a transactional data set that contains all the transactions occurring in online retail. Ecommerce data is information relating to the visitors and performance of an online shop. It's mostly used by marketers e.g. in understanding consumer behavior and enhancing conversion funnels.
Objective:¶
The objective is to find out the features which have the most information context to differentiate the positive class and negative class and also build a model to predict whether a customer will buy a product or not.
Dataset¶
The data contains information on web sessions of a customer:
- "Administrative", "Administrative Duration", "Informational", "Informational Duration", "Product Related" and "Product Related Duration": These represent the number of different types of pages visited by the visitor in that session and total time spent in each of these page categories.
- The values of these features are derived from the URL information of the pages visited by the user and updated in real-time when a user takes an action, e.g. moving from one page to another.
- The "Bounce Rate", "Exit Rate" and "Page Value" features represent the metrics measured by "Google Analytics" for each page in the e-commerce site.
Bounce Rate: The value of "Bounce Rate" feature for a web page refers to the percentage of visitors who enter the site from that page and then leave ("bounce") without triggering any other requests to the analytics server during that session.
Exit Rate: The value of "Exit Rate" feature for a specific web page is calculated as for all pageviews to the page, the percentage that was the last in the session.
Dataset has average bounce rates and exit rates for a page customer landed on.
Read more about Bounce Rate vs Exit Rate here
- Page Value: The "Page Value" feature represents the average value for a web page that a user visited before completing an e-commerce transaction.
Read more about Page Value here.
- Special Day: The "Special Day" feature indicates the closeness of the site visiting time to a specific special day (e.g. Mother’s Day, Valentine's Day) in which the sessions are more likely to be finalized with the transaction.
- The value of this attribute is determined by considering the dynamics of e-commerce such as the duration between the order date and delivery date.
- For example, for Valentina’s day, this value takes a nonzero value between February 2 and February 12, zero before and after this date unless it is close to another special day, and its maximum value of 1 on February 8.
The dataset also includes the operating system, browser, region, traffic type - these values are masked.
VisitorType: returning visitor, new visitor, or other types of customer.
- weekend: a Boolean value indicating whether the date of the visit is weekend or not
- month: month of the year
Import the necessary packages¶
In [1]:
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import warnings
from sklearn.model_selection import train_test_split
from sklearn import tree
from sklearn import metrics
from sklearn.tree import DecisionTreeClassifier
warnings.filterwarnings('ignore')
Read the dataset¶
In [2]:
shoppers = pd.read_csv('online_shoppers_intention.csv')
In [3]:
# copying data to another varaible to avoid any changes to original data
data=shoppers.copy()
View the first and last 10 rows of the dataset.¶
In [4]:
data.head(10)
Out[4]:
In [5]:
data.tail(10)
Out[5]:
Understand the shape of the dataset.¶
In [6]:
data.shape
Out[6]:
- Dataset has 12330 rows and 18 columns
Let's check the duplicate data. And if any, we should remove it.¶
In [7]:
data[data.duplicated()].count()
Out[7]:
Let's drop the duplicate values. As we can see from above that 125 are duplicates.¶
In [8]:
data.drop_duplicates(inplace=True)
Check the data types of the columns for the dataset.¶
In [9]:
data.info()
Insights:¶
- Most of the data-types are either int64 or float64.
- 2 columns - Month and VisitorType are having data-types as an object, this means we need to convert these into suitable data-type before we feed our data into the model.
- The last two columns: "Weekend" and "Revenue", have the data-type as a bool.
Think about it:¶
- We already know that we convert the categorical data-types into suitable form using either the "LabelEncoder" or "OneHotEncoding".
- But here we have a bool data type. What should we do here?
In Python, True and False are cast implicitly into integers:
True == 1False == 0This means that an algorithm running in pure Python should work without conversion.
Fixing the data types¶
- Month and VisitorType are of an object type, we can change them to categories.
converting "objects" to "category" reduces the data space required to store the dataframe
In [10]:
data["Month"] = data["Month"].astype("category")
data["VisitorType"] = data["VisitorType"].astype("category")
In [11]:
data.info()
- Month and VisitorType have been converted to categories.
we can see that the memory usage has decreased from 1.6+ MB to 1.4 MB
Check for missing values¶
In [12]:
data.isnull().sum()
Out[12]:
- There are no missing values in the data.
Give a statistical summary for the dataset.¶
In [13]:
data.describe().T
Out[13]:
- All the numerical variables seem to be right-skewed.
- Most time spent by customers is on the ProductRelated pages.
- There are some customers who haven't visited the administrative page and informational pages.
In [14]:
data.describe(include=['category','bool'])
Out[14]:
In [15]:
data['Month'].unique()
Out[15]:
In [16]:
data['VisitorType'].unique()
Out[16]:
In [17]:
data['Weekend'].unique()
Out[17]:
In [18]:
data['Revenue'].unique()
Out[18]:
- Data is of 10 months January and April's data is not available with us.
- In may the website had the most active customers.
- Most of the customers are returning type customers which is a good thing for the business.
- Most traffic on the website is generally on the weekdays.
- Website is only able to generate revenue from a small portion of customers.
EDA¶
Univariate analysis¶
In [19]:
# While doing uni-variate analysis of numerical variables we want to study their central tendency
# and dispersion.
# Let us write a function that will help us create boxplot and histogram for any input numerical
# variable.
# This function takes the numerical column as the input and returns the boxplots
# and histograms for the variable.
# Let us see if this help us write faster and cleaner code.
def histogram_boxplot(feature, figsize=(15,10), bins = None):
""" Boxplot and histogram combined
feature: 1-d feature array
figsize: size of fig (default (9,8))
bins: number of bins (default None / auto)
"""
f2, (ax_box2, ax_hist2) = plt.subplots(nrows = 2, # Number of rows of the subplot grid= 2
sharex = True, # x-axis will be shared among all subplots
gridspec_kw = {"height_ratios": (.25, .75)},
figsize = figsize
) # creating the 2 subplots
sns.boxplot(feature, ax=ax_box2, showmeans=True, color='violet') # boxplot will be created and a star will indicate the mean value of the column
sns.distplot(feature, kde=F, ax=ax_hist2, bins=bins,palette="winter") if bins else sns.distplot(feature, kde=False, ax=ax_hist2) # For histogram
ax_hist2.axvline(np.mean(feature), color='green', linestyle='--') # Add mean to the histogram
ax_hist2.axvline(np.median(feature), color='black', linestyle='-') # Add median to the histogram
Observations on Administrative_Duration¶
In [20]:
histogram_boxplot(data["Administrative_Duration"])
- The distribution of Administrative_Duration is right-skewed.
- There are outliers in this variable.
- From the boxplot we can see that the third quartile(Q3) is equal to 94 which means 75% of customers stay less than 94 seconds on Administrative pages in a session.
Observations on Informational¶
In [21]:
histogram_boxplot(data["Informational"])
- The distribution of Informational is right-skewed.
- There are outliers in this variable.
- Very few customers have visited the informational page.
Observations on Informational_Duration¶
In [22]:
histogram_boxplot(data["Informational_Duration"])
- The distribution of Informational_Duration is right-skewed.
- There are outliers in this variable.
- On average customers have spent 35 seconds on the informational page.
Observations on ProductRelated¶
In [23]:
histogram_boxplot(data["ProductRelated"])
- The distribution of ProductRelated is right-skewed.
- There are outliers in this variable.
- From the boxplot we can see that the third quartile(Q3) is equal to 38 which means 75% of customers have visited less than 38 pages and on average a customer visits 32 pages.
Observations on ProductRelated_Duration¶
In [24]:
histogram_boxplot(data["ProductRelated_Duration"])
- The distribution of ProductRelated_Duration is right-skewed.
- There are outliers in this variable.
- On average customers have spent 1206 seconds (~20 minutes) on the ProductRelated page which is way more than the administrative page and informational page.
Observations on BounceRates¶
In [25]:
histogram_boxplot(data["BounceRates"])
- The distribution of BounceRates is right-skewed.
- There are outliers in this variable.
- On average the bounce rate of a webpage is 0.022.
Observations on ExitRates¶
In [26]:
histogram_boxplot(data["ExitRates"])
- The distribution of ExitRates is right-skewed.
- There are outliers in this variable.
- On average the bounce rate of a webpage is 0.041.
Observations on PageValues¶
In [27]:
histogram_boxplot(data["PageValues"])
- The distribution of PageValues is right-skewed.
- There are outliers in this variable.
- On average the customers visit 6 pages before finally landing on the transaction page.
Observations on SpecialDay¶
In [28]:
histogram_boxplot(data["SpecialDay"])
- The distribution of SpecialDays suggests that most customers have visited the website on days which were not close to a special occasion.
- The distribution of SpecialDays also suggests we should look at it as a category to extract more information.
Observations on OperatingSystems¶
In [29]:
histogram_boxplot(data["OperatingSystems"])
- The distribution of Operating systems shows most of the customers accessed the website using 2nd operating system.
- These values are masked so it is difficult to comment on it.
- The distribution of Operating systems also suggests we should look at it as a category to extract more information.
Observations on Browser¶
In [30]:
histogram_boxplot(data["Browser"])
- The distribution of Browser shows most of the customers accessed the website using 2nd browser.
- These values are masked so it is difficult to comment on it.
- The distribution of Browser also suggests we should look at it as a category to extract more information.
Observations on Region¶
In [31]:
histogram_boxplot(data["Region"])
- The distribution of Region shows most of the customers accessed the website from Region 1.
- These values are masked so it is difficult to comment on it.
- The distribution of Region also suggests we should look at it as a category to extract more information.
Observations on TrafficType¶
In [32]:
histogram_boxplot(data["TrafficType"])
- The distribution of TrafficTyoes shows most of types of traffic the website gets is '2.
- These values are masked so it is difficult to comment on it.
- The distribution of Traffic also suggests we should look at it as a category to extract more information.
In [33]:
# Function to create barplots that indicate percentage for each category.
def perc_on_bar(plot, feature):
'''
plot
feature: categorical feature
the function won't work if a column is passed in hue parameter
'''
total = len(feature) # length of the column
for p in ax.patches:
percentage = '{:.1f}%'.format(100 * p.get_height()/total) # percentage of each class of the category
x = p.get_x() + p.get_width() / 2 - 0.05 # width of the plot
y = p.get_y() + p.get_height() # hieght of the plot
ax.annotate(percentage, (x, y), size = 12) # annotate the percantage
plt.show() # show the plot
Observations on Month¶
In [34]:
plt.figure(figsize=(15,5))
ax = sns.countplot(data["Month"],palette='winter',order=['Feb','Mar','May','June','Jul','Aug','Sep','Oct','Nov','Dec'])
perc_on_bar(ax,data["Month"])
- 27.3% of the users visited the website in the month of May followed by November with 24.4% of traffic.
Observations on VisitorType¶
In [35]:
plt.figure(figsize=(15,5))
ax = sns.countplot(data["VisitorType"],palette='winter')
perc_on_bar(ax,data["VisitorType"])
- 85.5% of the customer are returning customers, indicating the website has a loyal base of customers.
- During these 10 months website also say 13.9% of the customers visiting the website.
Observations on SpecialDay¶
In [36]:
plt.figure(figsize=(15,5))
ax = sns.countplot(data["SpecialDay"],palette='winter')
perc_on_bar(ax,data["SpecialDay"])
- 89.9% of the website sessions are on Non-Special days.
Observations on OperatingSystems¶
In [37]:
plt.figure(figsize=(15,5))
ax = sns.countplot(data["OperatingSystems"],palette='winter')
perc_on_bar(ax,data["OperatingSystems"])
- 53.6% of the customers use '2' operating system.
Observations on Region¶
In [38]:
plt.figure(figsize=(15,5))
ax = sns.countplot(data["Region"],palette='winter')
perc_on_bar(ax,data["Region"])
- 38.6% of the website sessions are from customers of Region 1.
Observations on TrafficType¶
In [39]:
plt.figure(figsize=(15,7))
ax = sns.countplot(data["TrafficType"],palette='winter')
perc_on_bar(ax,data["TrafficType"])
- 32% of the traffic on website is of type 2.
Observations on Weekend¶
In [40]:
plt.figure(figsize=(15,7))
ax = sns.countplot(data["Weekend"],palette='winter')
perc_on_bar(ax,data["Weekend"])
- 76.6% of the website sessions are on Non-weekend days.
Observations on Revenue¶
In [41]:
plt.figure(figsize=(15,7))
ax = sns.countplot(data["Revenue"],palette='winter')
perc_on_bar(ax,data["Revenue"])
- Website is able to generate revenue from only 15.6% of the customers.
Observations on Administrative¶
In [42]:
plt.figure(figsize=(15,7))
ax = sns.countplot(data["Administrative"],palette='winter')
perc_on_bar(ax,data["Administrative"])
- 46.2% of the customers have not visited the administrative page.
Observations on Informational¶
In [43]:
plt.figure(figsize=(15,7))
ax = sns.countplot(data["Informational"],palette='winter')
perc_on_bar(ax,data["Informational"])
- 78.4% of the customers have not visited the Informational page.
Observations on ProductRelated¶
In [44]:
plt.figure(figsize=(15,7))
ax = sns.countplot(data["ProductRelated"],palette='winter')
perc_on_bar(ax,data["ProductRelated"])
In [45]:
(len(data[data["ProductRelated"]==0]["ProductRelated"])/len(data))*100
Out[45]:
- Only 0.3% of customers have not visited the product related pages.
Bivariate Analysis¶
In [46]:
plt.figure(figsize=(15,7))
sns.heatmap(data.corr(),annot=True)
plt.show()
- Revenue shows the highest correlation with PageValues (0.49) simply because PageValues takes in account the pages visited before reaching the 'transaction' page.
- Administrative, Informational and ProductRelated pages are correlated with the Administrative, Informational and ProductRelated time durations spent on them which is understandable.
- BounceRates and ExitRates are very highly correlated(0.9) with each other.
In [47]:
sns.pairplot(data=data,hue="Revenue",)
plt.show()
- We can see varying distributions in variables for revenue, we should investigate it further.
Revenue vs Administrative, Informational and ProductRelated pages and time spent on these pages¶
In [48]:
cols = data[['Administrative','Administrative_Duration','Informational','Informational_Duration','ProductRelated','ProductRelated_Duration']].columns.tolist()
plt.figure(figsize=(12,7))
for i, variable in enumerate(cols):
plt.subplot(3,2,i+1)
sns.boxplot(data["Revenue"],data[variable],palette="PuBu")
plt.tight_layout()
plt.title(variable)
plt.show()
It is difficult to make an interpretation from the graphs above let's visualize them by removing these outliers (for visualization not from orignal data) to get a better understanding
In [49]:
cols = data[['Administrative','Administrative_Duration','Informational','Informational_Duration','ProductRelated','ProductRelated_Duration']].columns.tolist()
plt.figure(figsize=(12,7))
for i, variable in enumerate(cols):
plt.subplot(3,2,i+1)
sns.boxplot(data["Revenue"],data[variable],palette="PuBu",showfliers=False)
plt.tight_layout()
plt.title(variable)
plt.show()
- Customers who have visited Administrative, Informational and ProductRelated pages more times contribute to the revenue as compared to the one's who visited less pages.
- Those customers who spend more time on the Administrative, Informational and ProductRelated pages help in generating the revenue.
- But in both cases - visiting the page and spending more time on pages have many outliers.
- There is clear distinction between the customers - visiting more pages and spening more time on pages contributes to the revenue.
Revenue vs Bounce Rates,Exit Rates¶
In [50]:
cols = data[['BounceRates','ExitRates']].columns.tolist()
plt.figure(figsize=(10,5))
for i, variable in enumerate(cols):
plt.subplot(1,2,i+1)
sns.boxplot(data["Revenue"],data[variable],palette="PuBu")
plt.tight_layout()
plt.title(variable)
plt.show()
- It is logical that customers who leave the website withouth triggering any response on server(definition of bounce rate) wouldn't be contributing to the revenue, similarly for the exit rates.
Revenue vs PageValues¶
In [51]:
plt.figure(figsize=(10,5))
sns.boxplot(data['Revenue'],data['PageValues'])
plt.show()
- Higher PageValues means higher contribution to revenue.
Revenue vs SpecialDay¶
In [52]:
def stacked_plot(x):
sns.set(palette='nipy_spectral')
tab1 = pd.crosstab(x,data['Revenue'],margins=True)
print(tab1)
print('-'*120)
tab = pd.crosstab(x,data['Revenue'],normalize='index')
tab.plot(kind='bar',stacked=True,figsize=(10,5))
plt.legend(loc='lower left', frameon=False)
plt.legend(loc="upper left", bbox_to_anchor=(1,1))
plt.show()
In [53]:
stacked_plot(data['SpecialDay'])
- Regular days contribute more to the revenue instead of special occasions which is understandable as most days are non-special days, but there are very few conversion sessions on special days.
Revenue vs Month¶
In [54]:
stacked_plot(data["Month"])
- November and May are the months with largest contributions in the revenue.
- There are lesser sessions in November as compared to May but more conversions.
In [55]:
revenue_data = data[data['Revenue']==True]
revenue_data.groupby(['Month','SpecialDay'])['Revenue'].count()
Out[55]:
- Revenue sessions on or around Special day were only observed in May, this means website is not able to capatilize on the special occasions.
Revenue vs OperatingSystems¶
In [56]:
stacked_plot(data["OperatingSystems"])
- Custoemrs using operating system '2' are the highest contributors to revenue, approximately 20% (1155).
Revenue vs Browser¶
In [57]:
stacked_plot(data["Browser"])
- Approximately 20%(1223 customers) of cutomers using Browser '2' contribute to the revenue.
Revenue vs Region¶
In [58]:
stacked_plot(data["Region"])
- There is a similar distrbution in regions.
In [59]:
stacked_plot(data["TrafficType"])
- There is a varying pattern of revenue sessions among different traffic sources, although traffic coming source 2 has the highest proportion of revenue sessions.
Revenue vs VisitorType¶
In [60]:
stacked_plot(data["VisitorType"])
- Returning customers are the largest contributors to the revenue. But interesting New visitors end up having more conversion sessions.
Revenue vs Weekend¶
In [61]:
stacked_plot(data["Weekend"])
- There is not much difference in weekend and week-day sessions.
Customer-Level Analysis¶
In [62]:
cols = data[['Administrative','Administrative_Duration','Informational','Informational_Duration','ProductRelated','ProductRelated_Duration']].columns.tolist()
plt.figure(figsize=(10,10))
for i, variable in enumerate(cols):
plt.subplot(4,2,i+1)
sns.boxplot(data["VisitorType"],data[variable],hue=data['Revenue'],palette="PuBu", showfliers=False) #turning-off outliers
plt.tight_layout()
plt.legend(loc="upper left", bbox_to_anchor=(1,1))
plt.title(variable)
plt.show()
- New visitors who have visited more administrative pages and spent more time on them have not contributed to the revenue - This indicates that the administrative pages need to be user friendly and simplified.
- Returning customers visiting informational pages have contributed to revenue - The informational pages are doing well to provide customers with all the required information.
- ProductRelated pages are doing a fair job.
In [63]:
tab1 = pd.crosstab(data['Month'],data['VisitorType'],margins=True)
print(tab1)
print('-'*120)
tab = pd.crosstab(data['Month'],data['VisitorType'],normalize='index')
tab.plot(kind='bar',stacked=True,figsize=(10,5))
plt.legend(loc='lower left', frameon=False)
plt.legend(loc="upper left", bbox_to_anchor=(1,1))
plt.show()
- November and December saw most new customers, but the proportion of new customers to returning customers is highest in October and September.
Data Preperation¶
In [64]:
# page values have the information related to the transaction activity of a customer and would create a bias in model.
data = data.drop(['PageValues'],axis=1)
In [65]:
dummy_data = pd.get_dummies(data, columns=['Month','VisitorType','Weekend','Region','Browser',
'OperatingSystems','SpecialDay'],drop_first=True)
dummy_data.head()
Out[65]:
Model Building - Approach¶
- Data preparation
- Partition the data into train and test set.
- Built a CART model on the train data.
- Tune the model and prune the tree, if required.
- Test the data on test set.
In [66]:
column_names = list(dummy_data.columns)
column_names.remove('Revenue') # Keep only names of features by removing the name of target variable
feature_names = column_names
print(feature_names)
Split Data¶
In [67]:
X = dummy_data.drop('Revenue',axis=1) # Features
y = dummy_data['Revenue'].astype('int64') # Labels (Target Variable)
# converting target to integers - since some functions might not work with bool type
In [68]:
# Splitting data into training and test set:
X_train, X_test, y_train, y_test =train_test_split(X, y, test_size=0.3, random_state=1)
print(X_train.shape, X_test.shape)
Build Decision Tree Model¶
- We will build our model using the DecisionTreeClassifier function. Using default 'gini' criteria to split.
If the frequency of class A is 10% and the frequency of class B is 90%, then class B will become the dominant class and the decision tree will become biased toward the dominant classes.
In this case, we can pass a dictionary {0:0.15,1:0.85} to the model to specify the weight of each class and the decision tree will give more weightage to class 1.
class_weight is a hyperparameter for the decision tree classifier.
In [69]:
model = DecisionTreeClassifier(criterion='gini',class_weight={0:0.15,1:0.85},random_state=1)
In [70]:
model.fit(X_train, y_train)
Out[70]:
In [71]:
def make_confusion_matrix(model,y_actual,labels=[1, 0]):
'''
model : classifier to predict values of X
y_actual : ground truth
'''
y_predict = model.predict(X_test)
cm=metrics.confusion_matrix( y_actual, y_predict, labels=[0, 1])
df_cm = pd.DataFrame(cm, index = [i for i in ["Actual - No","Actual - Yes"]],
columns = [i for i in ['Predicted - No','Predicted - Yes']])
group_counts = ["{0:0.0f}".format(value) for value in
cm.flatten()]
group_percentages = ["{0:.2%}".format(value) for value in
cm.flatten()/np.sum(cm)]
labels = [f"{v1}\n{v2}" for v1, v2 in
zip(group_counts,group_percentages)]
labels = np.asarray(labels).reshape(2,2)
plt.figure(figsize = (10,7))
sns.heatmap(df_cm, annot=labels,fmt='')
plt.ylabel('True label')
plt.xlabel('Predicted label')
In [72]:
make_confusion_matrix(model,y_test)
In [73]:
y_train.value_counts(1)
Out[73]:
We only have 15% of positive classes, so if our model marks each sample as negative, then also we'll get 84% accuracy, hence accuracy is not a good metric to evaluate here.
Insights:¶
True Positives:
- Reality: A customer made a purchase.
- Model predicted: The customer will contribute to revenue.
- Outcome: The model is good.
True Negatives:
- Reality: A customer did NOT make a purchase.
- Model predicted: The customer will NOT contribute to revenue.
- Outcome: The business is unaffected.
False Positives:
- Reality: A customer did NOT make a purchase.
- Model predicted: The customer will contribute to revenue.
- Outcome: The team which is targeting the potential customers will be wasting their resources on the people/customers who will not be contributing to the revenue.
False Negatives:
- Reality: A customer made a purchase.
- Model predicted: The customer will NOT contribute to revenue.
- Outcome: The potential customer is missed by the sales/marketing team, the team could have offered the potential customer some discount or loyalty card to make the customer come again to purchase. (Customer retention will get affected.)
- In this case, not being able to identify a potential customer is the biggest loss we can face. Hence, recall is the right metric to check the performance of the model.
In [74]:
## Function to calculate recall score
def get_recall_score(model):
'''
model : classifier to predict values of X
'''
pred_train = model.predict(X_train)
pred_test = model.predict(X_test)
print("Recall on training set : ",metrics.recall_score(y_train,pred_train))
print("Recall on test set : ",metrics.recall_score(y_test,pred_test))
In [76]:
get_recall_score(model)
- There is a huge disparity in performance of model on training set and test set, which suggests that the model is overfiiting.
Visualizing the Decision Tree¶
In [77]:
plt.figure(figsize=(20,30))
out = tree.plot_tree(model,feature_names=feature_names,filled=True,fontsize=9,node_ids=False,class_names=None,)
#below code will add arrows to the decision tree split if they are missing
for o in out:
arrow = o.arrow_patch
if arrow is not None:
arrow.set_edgecolor('black')
arrow.set_linewidth(1)
plt.show()
In [78]:
# Text report showing the rules of a decision tree -
print(tree.export_text(model,feature_names=feature_names,show_weights=True))
In [79]:
# importance of features in the tree building ( The importance of a feature is computed as the
#(normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance )
print (pd.DataFrame(model.feature_importances_, columns = ["Imp"], index = X_train.columns).sort_values(by = 'Imp', ascending = False))
In [80]:
importances = model.feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12,12))
plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='violet', align='center')
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel('Relative Importance')
plt.show()
- According to the decision tree model, PageValues is the most important variable for predicting the Revenue.
The tree above is very complex and difficult to interpret.
Reducing over fitting¶
Using GridSearch for Hyperparameter tuning of our tree model¶
- Hyperparameter tuning is also tricky in the sense that there is no direct way to calculate how a change in the hyperparameter value will reduce the loss of your model, so we usually resort to experimentation. i.e we'll use Grid search
- Grid search is a tuning technique that attempts to compute the optimum values of hyperparameters.
- It is an exhaustive search that is performed on a the specific parameter values of a model.
- The parameters of the estimator/model used to apply these methods are optimized by cross-validated grid-search over a parameter grid.
In [81]:
from sklearn.model_selection import GridSearchCV
In [82]:
# Choose the type of classifier.
estimator = DecisionTreeClassifier(random_state=1,class_weight = {0:.15,1:.85})
# Grid of parameters to choose from
parameters = {
'max_depth': np.arange(1,10),
'criterion': ['entropy','gini'],
'splitter': ['best','random'],
'min_impurity_decrease': [0.000001,0.00001,0.0001],
'max_features': ['log2','sqrt']
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.recall_score)
# Run the grid search
grid_obj = GridSearchCV(estimator, parameters, scoring=scorer,cv=5)
grid_obj = grid_obj.fit(X_train, y_train)
# Set the clf to the best combination of parameters
estimator = grid_obj.best_estimator_
# Fit the best algorithm to the data.
estimator.fit(X_train, y_train)
Out[82]:
In [83]:
make_confusion_matrix(estimator,y_test)
In [84]:
get_recall_score(estimator)
Recall has improved for both train and test set after hyperparameter tuning and we have a generalized model.
Visualizing the Decision Tree¶
In [85]:
plt.figure(figsize=(15,10))
out = tree.plot_tree(estimator,feature_names=feature_names,filled=True,fontsize=9,node_ids=False,class_names=None)
for o in out:
arrow = o.arrow_patch
if arrow is not None:
arrow.set_edgecolor('black')
arrow.set_linewidth(1)
plt.show()
In [86]:
# Text report showing the rules of a decision tree -
print(tree.export_text(estimator,feature_names=feature_names,show_weights=True))
In [87]:
# importance of features in the tree building ( The importance of a feature is computed as the
#(normalized) total reduction of the 'criterion' brought by that feature. It is also known as the Gini importance )
print (pd.DataFrame(estimator.feature_importances_, columns = ["Imp"], index = X_train.columns).sort_values(by = 'Imp', ascending = False))
#Here we will see that importance of features has increased
In [88]:
importances = estimator.feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12,12))
plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='violet', align='center')
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel('Relative Importance')
plt.show()
Cost Complexity Pruning¶
The DecisionTreeClassifier provides parameters such as
min_samples_leaf and max_depth to prevent a tree from overfiting. Cost
complexity pruning provides another option to control the size of a tree. In
DecisionTreeClassifier, this pruning technique is parameterized by the
cost complexity parameter, ccp_alpha. Greater values of ccp_alpha
increase the number of nodes pruned. Here we only show the effect of
ccp_alpha on regularizing the trees and how to choose a ccp_alpha
based on validation scores.
Total impurity of leaves vs effective alphas of pruned tree¶
Minimal cost complexity pruning recursively finds the node with the "weakest
link". The weakest link is characterized by an effective alpha, where the
nodes with the smallest effective alpha are pruned first. To get an idea of
what values of ccp_alpha could be appropriate, scikit-learn provides
DecisionTreeClassifier.cost_complexity_pruning_path that returns the
effective alphas and the corresponding total leaf impurities at each step of
the pruning process. As alpha increases, more of the tree is pruned, which
increases the total impurity of its leaves.
In [89]:
clf = DecisionTreeClassifier(random_state=1,class_weight = {0:0.15,1:0.85})
path = clf.cost_complexity_pruning_path(X_train, y_train)
ccp_alphas, impurities = path.ccp_alphas, path.impurities
In [90]:
pd.DataFrame(path)
Out[90]:
In [91]:
fig, ax = plt.subplots(figsize=(10,5))
ax.plot(ccp_alphas[:-1], impurities[:-1], marker='o', drawstyle="steps-post")
ax.set_xlabel("effective alpha")
ax.set_ylabel("total impurity of leaves")
ax.set_title("Total Impurity vs effective alpha for training set")
plt.show()
Next, we train a decision tree using the effective alphas. The last value
in ccp_alphas is the alpha value that prunes the whole tree,
leaving the tree, clfs[-1], with one node.
In [92]:
clfs = []
for ccp_alpha in ccp_alphas:
clf = DecisionTreeClassifier(random_state=1, ccp_alpha=ccp_alpha,class_weight = {0:0.15,1:0.85})
clf.fit(X_train, y_train)
clfs.append(clf)
print("Number of nodes in the last tree is: {} with ccp_alpha: {}".format(
clfs[-1].tree_.node_count, ccp_alphas[-1]))
For the remainder, we remove the last element in
clfs and ccp_alphas, because it is the trivial tree with only one
node. Here we show that the number of nodes and tree depth decreases as alpha
increases.
In [93]:
clfs = clfs[:-1]
ccp_alphas = ccp_alphas[:-1]
node_counts = [clf.tree_.node_count for clf in clfs]
depth = [clf.tree_.max_depth for clf in clfs]
fig, ax = plt.subplots(2, 1,figsize=(10,7))
ax[0].plot(ccp_alphas, node_counts, marker='o', drawstyle="steps-post")
ax[0].set_xlabel("alpha")
ax[0].set_ylabel("number of nodes")
ax[0].set_title("Number of nodes vs alpha")
ax[1].plot(ccp_alphas, depth, marker='o', drawstyle="steps-post")
ax[1].set_xlabel("alpha")
ax[1].set_ylabel("depth of tree")
ax[1].set_title("Depth vs alpha")
fig.tight_layout()
In [94]:
recall_train=[]
for clf in clfs:
pred_train3=clf.predict(X_train)
values_train=metrics.recall_score(y_train,pred_train3)
recall_train.append(values_train)
In [95]:
recall_test=[]
for clf in clfs:
pred_test3=clf.predict(X_test)
values_test=metrics.recall_score(y_test,pred_test3)
recall_test.append(values_test)
In [96]:
train_scores = [clf.score(X_train, y_train) for clf in clfs]
test_scores = [clf.score(X_test, y_test) for clf in clfs]
In [97]:
fig, ax = plt.subplots(figsize=(15,5))
ax.set_xlabel("alpha")
ax.set_ylabel("Recall")
ax.set_title("Recall vs alpha for training and testing sets")
ax.plot(ccp_alphas, recall_train, marker='o', label="train",
drawstyle="steps-post",)
ax.plot(ccp_alphas, recall_test, marker='o', label="test",
drawstyle="steps-post")
ax.legend()
plt.show()
Maximum value of Recall is at 0.014 alpha, but if we choose decision tree will only have a root node and we would lose the buisness rules, instead we can choose alpha 0.002 retaining information and getting higher recall.
In [98]:
# creating the model where we get highest train and test recall
index_best_model = np.argmax(recall_test)
best_model = clfs[index_best_model]
print(best_model)
In [99]:
best_model.fit(X_train, y_train)
Out[99]:
In [100]:
make_confusion_matrix(best_model,y_test)
In [101]:
get_recall_score(best_model)
Visualizing the Decision Tree¶
In [102]:
plt.figure(figsize=(5,5))
out = tree.plot_tree(best_model,feature_names=feature_names,filled=True,fontsize=9,node_ids=False,class_names=None)
for o in out:
arrow = o.arrow_patch
if arrow is not None:
arrow.set_edgecolor('black')
arrow.set_linewidth(1)
plt.show()
- This model might be giving the highest recall but a buisness would not be able to use it to actually target the potential customers.
Creating model with 0.002 ccp_alpha
In [103]:
best_model2 = DecisionTreeClassifier(ccp_alpha=0.002,
class_weight={0: 0.15, 1: 0.85}, random_state=1)
best_model2.fit(X_train, y_train)
Out[103]:
In [104]:
make_confusion_matrix(best_model2,y_test)
- We are able to identify more True positives - customers that are actually going to contribute to revenue than any other model.
In [105]:
get_recall_score(best_model2)
- The results have improved from the initial model and we have got higher recall than the hyperparameter tuned model and generalized decision tree - having comparable performance on training and test set.
Visualizing the Decision Tree¶
In [106]:
plt.figure(figsize=(15,10))
out = tree.plot_tree(best_model2,feature_names=feature_names,filled=True,fontsize=9,node_ids=False,class_names=None)
for o in out:
arrow = o.arrow_patch
if arrow is not None:
arrow.set_edgecolor('black')
arrow.set_linewidth(1)
plt.show()
In [107]:
# Text report showing the rules of a decision tree -
print(tree.export_text(best_model2,feature_names=feature_names,show_weights=True))
In [108]:
# importance of features in the tree building ( The importance of a feature is computed as the
#(normalized) total reduction of the 'criterion' brought by that feature. It is also known as the Gini importance )
print (pd.DataFrame(best_model2.feature_importances_, columns = ["Imp"], index = X_train.columns).sort_values(by = 'Imp', ascending = False))
In [109]:
importances = best_model2.feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12,12))
plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='violet', align='center')
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel('Relative Importance')
plt.show()
- Exit rates and ProductRelated duration are the top two important features to predict customer sessions contributing to Revenue.
Comparing all the decision tree models¶
In [110]:
comparison_frame = pd.DataFrame({'Model':['Initial decision tree model','Decision treee with hyperparameter tuning',
'Decision tree with post-pruning'], 'Train_Recall':[1,0.83,0.89], 'Test_Recall':[0.30,0.81,0.86]})
comparison_frame
Out[110]:
Decision tree model with post pruning has given the best recall score on data.
Conclusions¶
- We analyzed the "Online Shoppers Purchasing Intention" using different techniques and used Decision Tree Classifier to build a predictive model for the same.
- The model built can be used to predict if a customer is going to contribute to Revenue generation (by purchasing) or not.
- We visualized different trees and their confusion matrix to get a better understanding of the model. Easy interpretation is one of the key benefits of Decision Trees.
- We verified the fact that how much less data preparation is needed for Decision Trees and such a simple model gave good results even with outliers and imbalanced classes which shows the robustness of Decision Trees.
- ExitRates, ProductRelated_Duration, Month_Nov and BounceRates are the most important variable in predicting the customers that will contribute to the revenue.
- We established the importance of hyper-parameters/ pruning to reduce overfitting.
Recommendations¶
According to the decision tree model -
a) If a customer lands on a page with an exit rate greater than 0.041 there's a very high chance the customer will not be contributing to the revenue.
b) If a customer lands on a page with an exit rate less than 0.041 and spends more than ~5 minutes on a product related page then there is a very high chance that the customer is going to buy something and contribute to the revenue.
It is observed that the more time the customers spend on Administrative, Informational and ProductRelated pages more is the likelihood of them contributing to the revenue. Although the website can not control the time spent by customers it can enhance the user experience on the website to engage them more on the website.
Browsing Customers - Employ the predictive model to predict potential customers (customers who can buy the product), Offer limited-time coupons/discounts on a real-time basis only to those customers. This can also be employed for the customers in months like March, May, November, and December, as in those months, the traffic is higher so these months have potential buying users.
It is observed that most of the traffic that the website sees is on the non-special days, While there is little to none traffic and revenue sessions on special days - the website should initiate schemes/offers on the special days to attract more customers on such days.
May and November were the months where the website saw the highest traffic with further data it should be investigated what portfolios were running in those months and an inspiration to create more such portfolios can be drawn and implemented.
Customer retention - Member Loyalty programs initiatives like special discounts, coupons, etc can be provided.
Better resource management - Regular days (Non-weekend) days is when the website sees the most traffic, resources such as customer care services can be allocated more for these days.
Website should be made more friendly, easily accessible, and feasible for other operating systems and browsers as new visitors seem to struggle with administrative pages.
Site Engagement - Version of the site for slower-internet users, Consistent and user-friendly mobile website design for more browsers and operating systems.